Kaggle Competition https://www.kaggle.com/c/web-traffic-time-series-forecasting

Using Kaggel competition as:

Data

Data sets are available here for download:
https://www.kaggle.com/c/web-traffic-time-series-forecasting/data

Data Structure (and Size)

Information based on full dataset for reference.

train:

  • 145K rows x 551 vars
  • each row has info for one article based on:
    • article title from URL
    • locale of wikipedia (en.wikipedia.org, zh.wikipedia.org, etc)
    • access type (all-access, desktop, etc)
    • agent (all-agents, spider, etc)
  • cols are dates in format X2015.08.02

key:

  • 8.7M rows x 2 vars
  • this file has article info with date appended and a corresponding id number
  • i believe only used for uploading data to Kaggle

Sampling

Due to the size of data, need to take a sample and work with that.

Sample can be based on:

  • random selection of rows (not so useful in this case)
  • selected pages
    • topic?
    • key words?
  • incorporate dimensions:
    • locale (en.wikipedia)
    • type of access (device?): eg. ‘desktop’
    • ‘agent’ (source?): eg. ‘spider’

First crack:

  • filter full train_1.csv for “en.wikipedia” and save
Error in eval(expr, envir, enclos) : object 'train1' not found
  • reduces down to 24k rows
  • still 50MB

Second crack:

  • select pages (including variations by user agent, etc):
    • Main page (‘Main_Page_en’)
    • Howard Hughes (‘Howard_Hughes_en’)
    • Orange is the New Black (‘Orange_is_the_New_Black_en’)
  • 15 rows
  • 54KB - much better for small computer

  • SAVE AND USE

subject <- read.csv("data-input/subject.csv", stringsAsFactors=FALSE)

WRANGLE SUBJECT DATA

Structure (extend for more date columns):

subject.temp <- subject[,c(1:3)]
str(subject.temp)
'data.frame':   12 obs. of  3 variables:
 $ Page       : chr  "Howard_Hughes_en.wikipedia.org_desktop_all-agents" "Main_Page_en.wikipedia.org_desktop_all-agents" "Orange_Is_the_New_Black_en.wikipedia.org_desktop_all-agents" "Howard_Hughes_en.wikipedia.org_all-access_spider" ...
 $ X2015.07.01: int  75357 11952559 28486 137 17207 168 109821 20381245 65947 34167 ...
 $ X2015.07.02: int  5396 12344021 26685 113 14756 132 13122 20752194 60189 7528 ...

Pages:

subject <- subject %>% arrange(Page)
package 㤼㸱bindrcpp㤼㸲 was built under R version 3.3.3
subject$Page
 [1] "Howard_Hughes_en.wikipedia.org_all-access_all-agents"          
 [2] "Howard_Hughes_en.wikipedia.org_all-access_spider"              
 [3] "Howard_Hughes_en.wikipedia.org_desktop_all-agents"             
 [4] "Howard_Hughes_en.wikipedia.org_mobile-web_all-agents"          
 [5] "Main_Page_en.wikipedia.org_all-access_all-agents"              
 [6] "Main_Page_en.wikipedia.org_all-access_spider"                  
 [7] "Main_Page_en.wikipedia.org_desktop_all-agents"                 
 [8] "Main_Page_en.wikipedia.org_mobile-web_all-agents"              
 [9] "Orange_Is_the_New_Black_en.wikipedia.org_all-access_all-agents"
[10] "Orange_Is_the_New_Black_en.wikipedia.org_all-access_spider"    
[11] "Orange_Is_the_New_Black_en.wikipedia.org_desktop_all-agents"   
[12] "Orange_Is_the_New_Black_en.wikipedia.org_mobile-web_all-agents"

3 components to Page:

  1. Subject
  2. Access: ‘all-access’, ‘desktop’, ‘mobile’
  3. Agent: ‘all-agents’, ‘spider’

Note that ‘all-access-all-agents’ is the total of the other variations.

Time Series Data Example

Try with one page variation.

main.all.all <- subject %>% filter(Page=="Main_Page_en.wikipedia.org_all-access_all-agents")
main.all.all.ts <- main.all.all %>% gather(key=date, value=views, -Page)
main.all.all.ts$date <- sub("X", "", main.all.all.ts$date) 
main.all.all.ts$date <- as.Date(main.all.all.ts$date, format="%Y.%m.%d")
main.all.all.ts$index <- 1:nrow(main.all.all.ts) ## index for time series data points
summary(main.all.all.ts)
     Page                date                views              index      
 Length:550         Min.   :2015-07-01   Min.   :13658940   Min.   :  1.0  
 Class :character   1st Qu.:2015-11-15   1st Qu.:18098508   1st Qu.:138.2  
 Mode  :character   Median :2016-03-31   Median :19457533   Median :275.5  
                    Mean   :2016-03-31   Mean   :21938511   Mean   :275.5  
                    3rd Qu.:2016-08-15   3rd Qu.:22212934   3rd Qu.:412.8  
                    Max.   :2016-12-31   Max.   :67264258   Max.   :550.0  
chart.title <- "Daily Views for Main page - all access, all agents"
plot.ts1 <- ggplot(main.all.all.ts, aes(x=date, y=views))+geom_line()+
  scale_y_continuous(labels=comma, expand=c(0,0))+theme_classic()+ggtitle(chart.title)
ggplotly(plot.ts1)
We recommend that you use the dev version of ggplot2 with `ggplotly()`
Install it with: `devtools::install_github('hadley/ggplot2')`

Time Series Modeling (single time series example)

Take the example of Main page, all access, all agent to build time series model based on single time series.

Approach 1: Linear Smoothing (Loess)

References:

Info:

  • Loess is short for ‘local regression’ - it is the most common method for smoothing volatile time series
  • Uses least squares regression on subsets of data (can control how finely grained the subsets are)
  • ggplot2 uses loess as default for geom_smooth when less than 1,000 data points
  • Loess using lots of memory and so is not suitable for huge data sets

Analyze (visualize) smoothing models

SAME CHART AS ABOVE WITH LOESS SMOOTHING ADDED (ggplot2 defaults)

plot.ts1+geom_smooth(method='loess')

  • Increase granularity with span (between 0-1, higher is smoother)
chart.title <- "Same Plot with loess span set lower for more granularity"
plot.ts1+geom_smooth(method='loess', span=0.3)

  • can layer in multiple loess smoothed lines for comparison (confidence intervals removed)
plot.ts1+
  geom_smooth(formula=y ~ x, method='loess', span=0.2, color='red', se=FALSE)+
  geom_smooth(method='loess', span=0.4, color='orange', se=FALSE)+
  geom_smooth(method='loess', span=0.6, color='green', se=FALSE)+
  geom_smooth(method='loess', span=0.8, color='purple', se=FALSE)+
  ggtitle("Same Plot with various smoothing lines (span adjusted, no conf. int.)")

Making a Prediction based on loess model

Get loess model from existing data

loess1 <- loess(views ~ as.numeric(date), data=main.all.all.ts, span=0.3)
summary(loess1)
Call:
loess(formula = views ~ as.numeric(date), data = main.all.all.ts, 
    span = 0.3)

Number of Observations: 550 
Equivalent Number of Parameters: 10.02 
Residual Standard Error: 5885000 
Trace of smoother matrix: 11.07  (exact)

Control settings:
  span     :  0.3 
  degree   :  2 
  family   :  gaussian
  surface  :  interpolate     cell = 0.2
  normalize:  TRUE
 parametric:  FALSE
drop.square:  FALSE 

Apply Loess model to future time periods

## extend date range for prediction period
ndays <- 30 ## number of days to predict
pred.period <- data.frame(date=seq(min(main.all.all.ts$date),max(main.all.all.ts$date+ndays), by='days'))
## prediction with loess1 doesn't work because default loess doesn't extrapolate 
#predict(loess1, pred.period, se=TRUE)
## new loess model: add control=...
sp <- 0.45 ## set span for model: lower number puts more weight on recent
loess2 <- loess(views ~ as.numeric(date), data=main.all.all.ts, control=loess.control(surface = 'direct'), span=sp)
## plot actual data with fitted data from model
plot.ts1+geom_line(aes(date, loess2$fitted, color='model'))

## predict with new loess model - extended period
pr <- predict(loess2, as.numeric(pred.period$date), se=TRUE)
## prediction (including existing data)
#pr[[1]] ## first object is prediction
## new data frame with dates and prediction
prediction <- pred.period %>% mutate(views.pred=pr[[1]])
## join date rate for prediction with existing data
main.pred <- left_join(prediction, main.all.all.ts, by='date') %>% select(-index)
## plot the result
chart.title <- "Daily Views for Main Page with Loess Model"
plot.ts2 <- ggplot(main.pred, aes(x=date, y=views))+geom_line()+
  scale_y_continuous(labels=comma, expand=c(0,0))+theme_classic()+
  ggtitle(chart.title)+geom_line(aes(date, views.pred, color='model+prediction'))
ggplotly(plot.ts2)
We recommend that you use the dev version of ggplot2 with `ggplotly()`
Install it with: `devtools::install_github('hadley/ggplot2')`

Span: 0.45 Number of days predicted: 30

Conclusion

  • Loess is great for smoothing a line to highlight general pattern
  • maybe not great for time series prediction - subject to manipulation

Approach 2

Reference: * http://r-statistics.co/Time-Series-Analysis-With-R.html

Convert data from data frame to time series object

  • Set up time series so that can apply time series functions for decomposition, etc
## time series forumlation examples from above reference
# ts (inputData, frequency = 4, start = c(1959, 2)) # frequency 4 => Quarterly Data
# ts (1:10, frequency = 12, start = 1990) # freq 12 => Monthly data. 
# ts (inputData, start=c(2009), end=c(2014), frequency=1) # Yearly Data
## See Notes below for explanation of frequency
ts.Main <- ts(main.all.all.ts$views, frequency=365, start=c(year(min(main.all.all.ts$date)), month(min(main.all.all.ts$date)), day(min(main.all.all.ts$date))))
ts.Main.all.all.wk <- ts(main.all.all.ts$views, frequency=52, start=c(year(min(main.all.all.ts$date)), month(min(main.all.all.ts$date)), day(min(main.all.all.ts$date))))

Notes:

  • since it is daily data, time series frequency=365 (days in yr)
  • however, because there is total 550 data points, not enough data for seasonality component to be estimated (need at least two iterations)
    • will get error: “time series has no or less than 2 periods”"
  • so…used frequency=52 to treat the time series as weekly data for illustration purposes
  • (great for illustrative purposes but might be hard to get a reliable prediction)

More on ‘time series has no or less than 2 periods’ error:
* https://stat.ethz.ch/pipermail/r-help/2013-October/361047.html

Decomposition

Using ‘decompose’

decomposedRes <- decompose(ts.Main.all.all.wk, type='additive') ## type='mult' if multiplicative; 'additive' if additive
plot(decomposedRes)

Using ‘stl’


stl.style <- stl(ts.Main.all.all.wk, s.window='periodic')
plot(stl.style)

Prediction

https://a-little-book-of-r-for-time-series.readthedocs.io/en/latest/

Reverting back to original data and forging ahead with prediction.

ts.Mainforecast
Holt-Winters exponential smoothing without trend and without seasonal component.

Call:
HoltWinters(x = ts.Main, beta = FALSE, gamma = FALSE)

Smoothing parameters:
 alpha: 0.9999487
 beta : FALSE
 gamma: FALSE

Coefficients:
      [,1]
a 26149449
  • alpha close to 1 means heavy weighting on recent (needs confirmation)

  • plotting forecast against actual

Forecast

LS0tDQp0aXRsZTogJ1RpbWUgU2VyaWVzIEZvcmVjYXN0aW5nIChiYXNlZCBvbiBLYWdnbGUgQ29tcGV0aXRpb246IFdlYiBUcmFmZmljIEZvcmVjYXN0aW5nKScNCm91dHB1dDoNCiAgaHRtbF9kb2N1bWVudDogZGVmYXVsdA0KICBodG1sX25vdGVib29rOiBkZWZhdWx0DQotLS0NCg0KS2FnZ2xlIENvbXBldGl0aW9uDQpodHRwczovL3d3dy5rYWdnbGUuY29tL2Mvd2ViLXRyYWZmaWMtdGltZS1zZXJpZXMtZm9yZWNhc3RpbmcgDQoNClVzaW5nIEthZ2dlbCBjb21wZXRpdGlvbiBhczoNCg0KKiBzb3VyY2Ugb2YgZGF0YSBpbiByZWxhdGVkIGZpZWxkDQoqIGxlYXJuIGZyb20gYXBwcm9hY2hlcyBieSBvdGhlcnMgKGtlcm5lbHMsIGV0YykNCg0KYGBge3IgaW5jbHVkZT1GQUxTRX0NCmtuaXRyOjpvcHRzX2NodW5rJHNldChlY2hvID0gVFJVRSwNCiAgICAgICAgICAgICAgICAgICAgICBtZXNzYWdlPUZBTFNFLA0KICAgICAgICAgICAgICAgICAgICAgIHdhcm5pbmc9RkFMU0UpDQojIyBsb2FkIHR5cGljYWwgbGlicmFyaWVzDQoNCmxpYnJhcnkodGlkeXZlcnNlKQ0KI2xpYnJhcnkoZHBseXIpDQojbGlicmFyeSh0aWR5cikNCiNsaWJyYXJ5KGdncGxvdDIpDQpsaWJyYXJ5KHBsb3RseSkNCmxpYnJhcnkoc2NhbGVzKQ0KbGlicmFyeShsdWJyaWRhdGUpDQpsaWJyYXJ5KGZvcmVjYXN0KQ0KDQpgYGANCiMjIERhdGENCg0KRGF0YSBzZXRzIGFyZSBhdmFpbGFibGUgaGVyZSBmb3IgZG93bmxvYWQ6IDxicj4NCmh0dHBzOi8vd3d3LmthZ2dsZS5jb20vYy93ZWItdHJhZmZpYy10aW1lLXNlcmllcy1mb3JlY2FzdGluZy9kYXRhDQoNCiogaW5pdGlhbCBkYXRhIGRvd25sb2FkZWQgZnJvbSBLYWdnbGUgYW5kIHNhdmVkIGluICdkYXRhLWlucHV0JyBmb2xkZXI6DQogICAgKyB0cmFpbl8xLmNzdiAoMjY1TUIpDQogICAgKyBrZXlfMS5jc3YgKDcxMU1CKQ0KICAgIA0KKiBkdWUgdG8gc2l6ZSBvZiBkYXRhc2V0LCBub3QgdXBsb2FkZWQgdG8gZ2l0IHJlcG9zaXRvcnkNCiogd29yayB3aXRoIHNhbXBsZSBpbnN0ZWFkIG9mIGZ1bGwgZGF0YSBzZXQNCg0KDQpgYGB7ciBJTVBPUlQgQU5EIElOU1BFQ1QgRlVMTCBEQVRBIFNFVCwgZWNobz1GQUxTRX0NCiMjIGlmIG5lY2Vzc2FyeSwgZGF0YSBjYW4gYmUgZG93bmxvYWRlZCBhbmQgaW1wb3J0ZWQNCiMjIHByZWZlciB0byB1c2Ugc21hbGwgc2FtcGxlIGFzIGV4cGxhaW5lZCBmdXJ0aGVyIGRvd24gdGhlIGZpbGUNCiMjIHdhcm5pbmc6IHRoaXMgY2FuIHRha2UgYSBsb25nIHRpbWUgZHVlIHRvIGZpbGUgc2l6ZSAoMTAgbWluIG9yIHNvIG9uIE1TIFN1cmZhY2UpDQojIHRyYWluMSA8LSByZWFkLmNzdigiZGF0YS1pbnB1dC90cmFpbl8xLmNzdiIsIHN0cmluZ3NBc0ZhY3RvcnMgPSBGQUxTRSkNCiMga2V5MSA8LSByZWFkLmNzdigiZGF0YS1pbnB1dC9rZXlfMS5jc3YiLCBzdHJpbmdzQXNGYWN0b3JzID0gRkFMU0UpDQojIA0KIyBoZWFkKGtleTEpDQojIGhlYWQodHJhaW4xKQ0KYGBgDQoNCiMjIyBEYXRhIFN0cnVjdHVyZSAoYW5kIFNpemUpDQoNCkluZm9ybWF0aW9uIGJhc2VkIG9uIGZ1bGwgZGF0YXNldCBmb3IgcmVmZXJlbmNlLg0KDQoqKnRyYWluOioqDQoNCiogMTQ1SyByb3dzIHggNTUxIHZhcnMNCiogZWFjaCByb3cgaGFzIGluZm8gZm9yIG9uZSBhcnRpY2xlIGJhc2VkIG9uOg0KICAgICsgYXJ0aWNsZSB0aXRsZSBmcm9tIFVSTA0KICAgICsgbG9jYWxlIG9mIHdpa2lwZWRpYSAoZW4ud2lraXBlZGlhLm9yZywgemgud2lraXBlZGlhLm9yZywgZXRjKQ0KICAgICsgYWNjZXNzIHR5cGUgKGFsbC1hY2Nlc3MsIGRlc2t0b3AsIGV0YykNCiAgICArIGFnZW50IChhbGwtYWdlbnRzLCBzcGlkZXIsIGV0YykNCiogY29scyBhcmUgZGF0ZXMgaW4gZm9ybWF0IFgyMDE1LjA4LjAyDQoNCioqa2V5OioqDQoNCiogOC43TSByb3dzIHggMiB2YXJzDQoqIHRoaXMgZmlsZSBoYXMgYXJ0aWNsZSBpbmZvIHdpdGggZGF0ZSBhcHBlbmRlZCBhbmQgYSBjb3JyZXNwb25kaW5nIGlkIG51bWJlcg0KKiBpIGJlbGlldmUgb25seSB1c2VkIGZvciB1cGxvYWRpbmcgZGF0YSB0byBLYWdnbGUNCg0KIyMjIyBTYW1wbGluZw0KDQpEdWUgdG8gdGhlIHNpemUgb2YgZGF0YSwgbmVlZCB0byB0YWtlIGEgc2FtcGxlIGFuZCB3b3JrIHdpdGggdGhhdC4NCg0KU2FtcGxlIGNhbiBiZSBiYXNlZCBvbjoNCg0KKiByYW5kb20gc2VsZWN0aW9uIG9mIHJvd3MgKG5vdCBzbyB1c2VmdWwgaW4gdGhpcyBjYXNlKQ0KKiBzZWxlY3RlZCBwYWdlcyANCiAgICArIHRvcGljPw0KICAgICsga2V5IHdvcmRzPw0KKiBpbmNvcnBvcmF0ZSBkaW1lbnNpb25zOg0KICAgICsgbG9jYWxlIChlbi53aWtpcGVkaWEpDQogICAgKyB0eXBlIG9mIGFjY2VzcyAoZGV2aWNlPyk6IGVnLiAnZGVza3RvcCcNCiAgICArICdhZ2VudCcgKHNvdXJjZT8pOiBlZy4gJ3NwaWRlcicNCiAgICANCkZpcnN0IGNyYWNrOg0KDQoqIGZpbHRlciBmdWxsIHRyYWluXzEuY3N2IGZvciAiZW4ud2lraXBlZGlhIiBhbmQgc2F2ZQ0KDQpgYGB7ciwgZWNobz1GQUxTRX0NCiMjIHN0YXJ0IHdpdGggZmlsdGVyIGZvciBlbg0KDQojIHRyYWluMS5lbiA8LSB0cmFpbjEgJT4lIGZpbHRlcihncmVwbCgiZW4ud2lraXBlZGlhIiwgdHJhaW4xJFBhZ2UsIGlnbm9yZS5jYXNlPVRSVUUpKQ0KIyAjIGxvb2tzIGdvb2QgLSBzYXZlIGZvciBmdXR1cmUgdXNlDQojIHdyaXRlLmNzdih0cmFpbjEuZW4sICJkYXRhLWlucHV0L3RyYWluLWVuLmNzdiIsIHJvdy5uYW1lcz1GQUxTRSkNCiMgDQojIHRyYWluLmVuIDwtIHJlYWQuY3N2KCJkYXRhLWlucHV0L3RyYWluLWVuLmNzdiIsIHN0cmluZ3NBc0ZhY3RvcnMgPSBGQUxTRSkNCmBgYA0KKiByZWR1Y2VzIGRvd24gdG8gMjRrIHJvd3MNCiogc3RpbGwgNTBNQg0KDQpTZWNvbmQgY3JhY2s6DQoNCiogc2VsZWN0IHBhZ2VzIChpbmNsdWRpbmcgdmFyaWF0aW9ucyBieSB1c2VyIGFnZW50LCBldGMpOg0KICAgICsgTWFpbiBwYWdlICgnTWFpbl9QYWdlX2VuJykNCiAgICArIEhvd2FyZCBIdWdoZXMgKCdIb3dhcmRfSHVnaGVzX2VuJykNCiAgICArIE9yYW5nZSBpcyB0aGUgTmV3IEJsYWNrICgnT3JhbmdlX2lzX3RoZV9OZXdfQmxhY2tfZW4nKQ0KICAgIA0KYGBge3IgU1VCSkVDVCBGSUxURVIgQU5EIFNBVkUsIGVjaG89RkFMU0V9DQojIHRyYWluLnN1YmplY3QgPC0gdHJhaW4uZW4gJT4lDQojICAgZmlsdGVyKGdyZXBsKCJeTWFpbl9QYWdlX2VufEhvd2FyZF9IdWdoZXNfZW58T3JhbmdlX0lzX3RoZV9OZXdfQmxhY2tfZW4iLCB0cmFpbi5lbiRQYWdlLCBpZ25vcmUuY2FzZT1GQUxTRSkpDQojIA0KIyB3cml0ZS5jc3YodHJhaW4uc3ViamVjdCwgImRhdGEtaW5wdXQvc3ViamVjdC5jc3YiLCByb3cubmFtZXMgPSBGQUxTRSkNCmBgYA0KDQoqIDE1IHJvd3MNCiogNTRLQiAtIG11Y2ggYmV0dGVyIGZvciBzbWFsbCBjb21wdXRlcg0KDQoqIFNBVkUgQU5EIFVTRQ0KDQpgYGB7ciBTVUJKRUNUIElNUE9SVH0NCnN1YmplY3QgPC0gcmVhZC5jc3YoImRhdGEtaW5wdXQvc3ViamVjdC5jc3YiLCBzdHJpbmdzQXNGYWN0b3JzPUZBTFNFKQ0KYGBgDQoNCiMjIyBXUkFOR0xFIFNVQkpFQ1QgREFUQQ0KDQpTdHJ1Y3R1cmUgKGV4dGVuZCBmb3IgbW9yZSBkYXRlIGNvbHVtbnMpOg0KDQpgYGB7ciBXUkFOR0xFIFNVQkpFQ1R9DQpzdWJqZWN0LnRlbXAgPC0gc3ViamVjdFssYygxOjMpXQ0Kc3RyKHN1YmplY3QudGVtcCkNCmBgYA0KDQpQYWdlczogDQoNCmBgYHtyfQ0Kc3ViamVjdCA8LSBzdWJqZWN0ICU+JSBhcnJhbmdlKFBhZ2UpDQpzdWJqZWN0JFBhZ2UNCg0KYGBgDQoNCjMgY29tcG9uZW50cyB0byBQYWdlOg0KDQoxLiBTdWJqZWN0DQoyLiBBY2Nlc3M6ICdhbGwtYWNjZXNzJywgJ2Rlc2t0b3AnLCAnbW9iaWxlJw0KMy4gQWdlbnQ6ICdhbGwtYWdlbnRzJywgJ3NwaWRlcicNCg0KTm90ZSB0aGF0ICdhbGwtYWNjZXNzLWFsbC1hZ2VudHMnIGlzIHRoZSB0b3RhbCBvZiB0aGUgb3RoZXIgdmFyaWF0aW9ucy4NCg0KIyMjIFRpbWUgU2VyaWVzIERhdGEgRXhhbXBsZQ0KDQpUcnkgd2l0aCBvbmUgcGFnZSB2YXJpYXRpb24uDQoNCmBgYHtyIE1BSU4gQUxMIENMRUFOfQ0KbWFpbi5hbGwuYWxsIDwtIHN1YmplY3QgJT4lIGZpbHRlcihQYWdlPT0iTWFpbl9QYWdlX2VuLndpa2lwZWRpYS5vcmdfYWxsLWFjY2Vzc19hbGwtYWdlbnRzIikNCm1haW4uYWxsLmFsbC50cyA8LSBtYWluLmFsbC5hbGwgJT4lIGdhdGhlcihrZXk9ZGF0ZSwgdmFsdWU9dmlld3MsIC1QYWdlKQ0KbWFpbi5hbGwuYWxsLnRzJGRhdGUgPC0gc3ViKCJYIiwgIiIsIG1haW4uYWxsLmFsbC50cyRkYXRlKSANCm1haW4uYWxsLmFsbC50cyRkYXRlIDwtIGFzLkRhdGUobWFpbi5hbGwuYWxsLnRzJGRhdGUsIGZvcm1hdD0iJVkuJW0uJWQiKQ0KbWFpbi5hbGwuYWxsLnRzJGluZGV4IDwtIDE6bnJvdyhtYWluLmFsbC5hbGwudHMpICMjIChub3QgbmVlZGVkKSBpbmRleCBmb3IgdGltZSBzZXJpZXMgZGF0YSBwb2ludHMNCg0Kc3VtbWFyeShtYWluLmFsbC5hbGwudHMpDQoNCndyaXRlX2NzdihtYWluLmFsbC5hbGwudHMsICJkYXRhLWlucHV0L21haW4tYWxsLmNzdiIpDQptYWluLmFsbC5hbGwudHMgPC0gcmVhZF9jc3YoImRhdGEtaW5wdXQvbWFpbi1hbGwuY3N2IikgIyMgaW5jbHVkZXMgaW5kZXggYWx0aG91Z2ggbm90IHVzZWQNCmBgYA0KDQpgYGB7ciBQTE9UIE1BSU4gQUxMfQ0KY2hhcnQudGl0bGUgPC0gIkRhaWx5IFZpZXdzIGZvciBNYWluIHBhZ2UgLSBhbGwgYWNjZXNzLCBhbGwgYWdlbnRzIg0KcGxvdC50czEgPC0gZ2dwbG90KG1haW4uYWxsLmFsbC50cywgYWVzKHg9ZGF0ZSwgeT12aWV3cykpK2dlb21fbGluZSgpKw0KICBzY2FsZV95X2NvbnRpbnVvdXMobGFiZWxzPWNvbW1hLCBleHBhbmQ9YygwLDApKSt0aGVtZV9jbGFzc2ljKCkrZ2d0aXRsZShjaGFydC50aXRsZSkNCmdncGxvdGx5KHBsb3QudHMxKQ0KYGBgDQoNCiMjIFRpbWUgU2VyaWVzIE1vZGVsaW5nIChzaW5nbGUgdGltZSBzZXJpZXMgZXhhbXBsZSkNCg0KVGFrZSB0aGUgZXhhbXBsZSBvZiBNYWluIHBhZ2UsIGFsbCBhY2Nlc3MsIGFsbCBhZ2VudCB0byBidWlsZCB0aW1lIHNlcmllcyBtb2RlbCBiYXNlZCBvbiBzaW5nbGUgdGltZSBzZXJpZXMuDQoNCiMjIyBBcHByb2FjaCAxOiBMaW5lYXIgU21vb3RoaW5nIChMb2VzcykNCg0KUmVmZXJlbmNlczoNCg0KKiBodHRwOi8vci1zdGF0aXN0aWNzLmNvL0xvZXNzLVJlZ3Jlc3Npb24tV2l0aC1SLmh0bWwgKGNvZGUgdG93YXJkIGJvdHRvbSBkb2Vzbid0IHdvcmspDQoqIGh0dHA6Ly9nZ3Bsb3QyLnRpZHl2ZXJzZS5vcmcvcmVmZXJlbmNlL2dlb21fc21vb3RoLmh0bWwNCg0KSW5mbzoNCg0KKiBMb2VzcyBpcyBzaG9ydCBmb3IgJ2xvY2FsIHJlZ3Jlc3Npb24nIC0gaXQgaXMgdGhlIG1vc3QgY29tbW9uIG1ldGhvZCBmb3Igc21vb3RoaW5nIHZvbGF0aWxlIHRpbWUgc2VyaWVzDQoqIFVzZXMgbGVhc3Qgc3F1YXJlcyByZWdyZXNzaW9uIG9uIHN1YnNldHMgb2YgZGF0YSAoY2FuIGNvbnRyb2wgaG93IGZpbmVseSBncmFpbmVkIHRoZSBzdWJzZXRzIGFyZSkNCiogZ2dwbG90MiB1c2VzIGxvZXNzIGFzIGRlZmF1bHQgZm9yIGdlb21fc21vb3RoIHdoZW4gbGVzcyB0aGFuIDEsMDAwIGRhdGEgcG9pbnRzDQoqIExvZXNzIHVzaW5nIGxvdHMgb2YgbWVtb3J5IGFuZCBzbyBpcyBub3Qgc3VpdGFibGUgZm9yIGh1Z2UgZGF0YSBzZXRzDQoNCiMjIyMgQW5hbHl6ZSAodmlzdWFsaXplKSBzbW9vdGhpbmcgbW9kZWxzDQoNClNBTUUgQ0hBUlQgQVMgQUJPVkUgV0lUSCBMT0VTUyBTTU9PVEhJTkcgQURERUQgKGdncGxvdDIgZGVmYXVsdHMpDQoNCmBgYHtyIFBMT1QgTUFJTiBBTEwgU01PT1RIfQ0KY2hhcnQudGl0bGUgPC0gIkRhaWx5IFZpZXdzIHdpdGggTG9lc3MgU21vb3RoZWQgTGluZSINCnBsb3QudHMxK2dlb21fc21vb3RoKG1ldGhvZD0nbG9lc3MnKStnZ3RpdGxlKGNoYXJ0LnRpdGxlKQ0KYGBgDQoNCiogSW5jcmVhc2UgZ3JhbnVsYXJpdHkgd2l0aCBzcGFuIChiZXR3ZWVuIDAtMSwgaGlnaGVyIGlzIHNtb290aGVyKQ0KDQpgYGB7ciBQTE9UIE1BSU4gQUxMIFNNT09USCAyfQ0KY2hhcnQudGl0bGUgPC0gIlNhbWUgUGxvdCB3aXRoIGxvZXNzIHNwYW4gc2V0IGxvd2VyIGZvciBtb3JlIGdyYW51bGFyaXR5Ig0KcGxvdC50czErZ2VvbV9zbW9vdGgobWV0aG9kPSdsb2VzcycsIHNwYW49MC4zKStnZ3RpdGxlKGNoYXJ0LnRpdGxlKQ0KYGBgDQoNCiogY2FuIGxheWVyIGluIG11bHRpcGxlIGxvZXNzIHNtb290aGVkIGxpbmVzIGZvciBjb21wYXJpc29uIChjb25maWRlbmNlIGludGVydmFscyByZW1vdmVkKQ0KDQpgYGB7ciBBTFRFUk5BVElWRSBTTU9PVEhFRCBMSU5FU30NCmNoYXJ0LnRpdGxlIDwtICJTYW1lIFBsb3Qgd2l0aCB2YXJpb3VzIHNtb290aGluZyBsaW5lcyAoc3BhbiBhZGp1c3RlZCwgbm8gY29uZi4gaW50LikiDQpwbG90LnRzMSsNCiAgZ2VvbV9zbW9vdGgoZm9ybXVsYT15IH4geCwgbWV0aG9kPSdsb2VzcycsIHNwYW49MC4yLCBjb2xvcj0ncmVkJywgc2U9RkFMU0UpKw0KICBnZW9tX3Ntb290aChtZXRob2Q9J2xvZXNzJywgc3Bhbj0wLjQsIGNvbG9yPSdvcmFuZ2UnLCBzZT1GQUxTRSkrDQogIGdlb21fc21vb3RoKG1ldGhvZD0nbG9lc3MnLCBzcGFuPTAuNiwgY29sb3I9J2dyZWVuJywgc2U9RkFMU0UpKw0KICBnZW9tX3Ntb290aChtZXRob2Q9J2xvZXNzJywgc3Bhbj0wLjgsIGNvbG9yPSdwdXJwbGUnLCBzZT1GQUxTRSkrDQogIGdndGl0bGUoY2hhcnQudGl0bGUpDQoNCmBgYA0KDQojIyMjIE1ha2luZyBhIFByZWRpY3Rpb24gYmFzZWQgb24gbG9lc3MgbW9kZWwNCg0KR2V0IGxvZXNzIG1vZGVsIGZyb20gZXhpc3RpbmcgZGF0YQ0KDQpgYGB7ciBMT0VTUyBNT0RFTH0NCiMjIExvZXNzIG1vZGVsDQojIyBjYW4gdXNlICdpbmRleCcgb3IganVzdCBjb252ZXJ0IGRhdGUgdG8gbnVtZXJpYw0KbG9lc3MxIDwtIGxvZXNzKHZpZXdzIH4gYXMubnVtZXJpYyhkYXRlKSwgZGF0YT1tYWluLmFsbC5hbGwudHMsIHNwYW49MC4zKQ0Kc3VtbWFyeShsb2VzczEpDQoNCmBgYA0KDQpBcHBseSBMb2VzcyBtb2RlbCB0byBmdXR1cmUgdGltZSBwZXJpb2RzDQoNCmBgYHtyIFBSRURJQ0lUT04gV0lUSCBMT0VTU30NCiMjIGV4dGVuZCBkYXRlIHJhbmdlIGZvciBwcmVkaWN0aW9uIHBlcmlvZA0KbmRheXMgPC0gNDUgIyMgbnVtYmVyIG9mIGRheXMgdG8gcHJlZGljdA0KcHJlZC5wZXJpb2QgPC0gZGF0YS5mcmFtZShkYXRlPXNlcShtaW4obWFpbi5hbGwuYWxsLnRzJGRhdGUpLG1heChtYWluLmFsbC5hbGwudHMkZGF0ZStuZGF5cyksIGJ5PSdkYXlzJykpDQoNCiMjIHByZWRpY3Rpb24gd2l0aCBsb2VzczEgZG9lc24ndCB3b3JrIGJlY2F1c2UgZGVmYXVsdCBsb2VzcyBkb2Vzbid0IGV4dHJhcG9sYXRlIA0KI3ByZWRpY3QobG9lc3MxLCBwcmVkLnBlcmlvZCwgc2U9VFJVRSkNCg0KIyMgbmV3IGxvZXNzIG1vZGVsOiBhZGQgY29udHJvbD0uLi4NCnNwIDwtIDAuNDUgIyMgc2V0IHNwYW4gZm9yIG1vZGVsOiBsb3dlciBudW1iZXIgcHV0cyBtb3JlIHdlaWdodCBvbiByZWNlbnQNCmxvZXNzMiA8LSBsb2Vzcyh2aWV3cyB+IGFzLm51bWVyaWMoZGF0ZSksIGRhdGE9bWFpbi5hbGwuYWxsLnRzLCBjb250cm9sPWxvZXNzLmNvbnRyb2woc3VyZmFjZSA9ICdkaXJlY3QnKSwgc3Bhbj1zcCkNCg0KIyMgcGxvdCBhY3R1YWwgZGF0YSB3aXRoIGZpdHRlZCBkYXRhIGZyb20gbW9kZWwNCmNoYXJ0LnRpdGxlIDwtICJEYWlseSBWaWV3cyB3aXRoIExvZXNzIE1vZGVsIGZpdHRlZCINCnBsb3QudHMxK2dlb21fbGluZShhZXMoZGF0ZSwgbG9lc3MyJGZpdHRlZCwgY29sb3I9J21vZGVsJykpK2dndGl0bGUoY2hhcnQudGl0bGUpDQoNCiMjIHByZWRpY3Qgd2l0aCBuZXcgbG9lc3MgbW9kZWwgLSBleHRlbmRlZCBwZXJpb2QNCnByIDwtIHByZWRpY3QobG9lc3MyLCBhcy5udW1lcmljKHByZWQucGVyaW9kJGRhdGUpLCBzZT1UUlVFKQ0KIyMgcHJlZGljdGlvbiAoaW5jbHVkaW5nIGV4aXN0aW5nIGRhdGEpDQojcHJbWzFdXSAjIyBmaXJzdCBvYmplY3QgaXMgcHJlZGljdGlvbg0KDQojIyBuZXcgZGF0YSBmcmFtZSB3aXRoIGRhdGVzIGFuZCBwcmVkaWN0aW9uDQpwcmVkaWN0aW9uIDwtIHByZWQucGVyaW9kICU+JSBtdXRhdGUodmlld3MucHJlZD1wcltbMV1dKQ0KDQojIyBqb2luIGRhdGUgcmF0ZSBmb3IgcHJlZGljdGlvbiB3aXRoIGV4aXN0aW5nIGRhdGENCm1haW4ucHJlZCA8LSBsZWZ0X2pvaW4ocHJlZGljdGlvbiwgbWFpbi5hbGwuYWxsLnRzLCBieT0nZGF0ZScpICU+JSBzZWxlY3QoLWluZGV4KQ0KDQojIyBwbG90IHRoZSByZXN1bHQNCmNoYXJ0LnRpdGxlIDwtICJEYWlseSBWaWV3cyBmb3IgTWFpbiBQYWdlIHdpdGggTG9lc3MgTW9kZWwgUHJlZGljdGlvbiINCnBsb3QudHMyIDwtIGdncGxvdChtYWluLnByZWQsIGFlcyh4PWRhdGUsIHk9dmlld3MpKStnZW9tX2xpbmUoKSsNCiAgc2NhbGVfeV9jb250aW51b3VzKGxhYmVscz1jb21tYSwgZXhwYW5kPWMoMCwwKSkrdGhlbWVfY2xhc3NpYygpKw0KICBnZ3RpdGxlKGNoYXJ0LnRpdGxlKStnZW9tX2xpbmUoYWVzKGRhdGUsIHZpZXdzLnByZWQsIGNvbG9yPSdtb2RlbCtwcmVkaWN0aW9uJykpDQpnZ3Bsb3RseShwbG90LnRzMikNCg0KYGBgDQoNClNwYW46IGByIHNwYA0KTnVtYmVyIG9mIGRheXMgcHJlZGljdGVkOiBgciBuZGF5c2ANCg0KIyMjIyBDb25jbHVzaW9uDQoNCiogTG9lc3MgaXMgZ3JlYXQgZm9yIHNtb290aGluZyBhIGxpbmUgdG8gaGlnaGxpZ2h0IGdlbmVyYWwgcGF0dGVybg0KKiBtYXliZSBub3QgZ3JlYXQgZm9yIHRpbWUgc2VyaWVzIHByZWRpY3Rpb24gLSBzdWJqZWN0IHRvIG1hbmlwdWxhdGlvbg0KDQojIyMgQXBwcm9hY2ggMg0KDQpSZWZlcmVuY2U6DQoqIGh0dHA6Ly9yLXN0YXRpc3RpY3MuY28vVGltZS1TZXJpZXMtQW5hbHlzaXMtV2l0aC1SLmh0bWwNCg0KIyMjIyBDb252ZXJ0IGRhdGEgZnJvbSBkYXRhIGZyYW1lIHRvIHRpbWUgc2VyaWVzIG9iamVjdA0KDQoqIFNldCB1cCB0aW1lIHNlcmllcyBzbyB0aGF0IGNhbiBhcHBseSB0aW1lIHNlcmllcyBmdW5jdGlvbnMgZm9yIGRlY29tcG9zaXRpb24sIGV0Yw0KDQpgYGB7ciBUSU1FIFNFUklFUyBNQUlOfQ0KIyMgdGltZSBzZXJpZXMgZm9ydW1sYXRpb24gZXhhbXBsZXMgZnJvbSBhYm92ZSByZWZlcmVuY2UNCiMgdHMgKGlucHV0RGF0YSwgZnJlcXVlbmN5ID0gNCwgc3RhcnQgPSBjKDE5NTksIDIpKSAjIGZyZXF1ZW5jeSA0ID0+IFF1YXJ0ZXJseSBEYXRhDQojIHRzICgxOjEwLCBmcmVxdWVuY3kgPSAxMiwgc3RhcnQgPSAxOTkwKSAjIGZyZXEgMTIgPT4gTW9udGhseSBkYXRhLiANCiMgdHMgKGlucHV0RGF0YSwgc3RhcnQ9YygyMDA5KSwgZW5kPWMoMjAxNCksIGZyZXF1ZW5jeT0xKSAjIFllYXJseSBEYXRhDQoNCiMjIENvbnZlcnQgZGF0YSBmcmFtZSB0byB0aW1lIHNlcmllczogZGFpbHkgZnJlcXVlbmN5DQp0cy5NYWluIDwtIHRzKG1haW4uYWxsLmFsbC50cyR2aWV3cywgZnJlcXVlbmN5PTM2NSwgc3RhcnQ9Yyh5ZWFyKG1pbihtYWluLmFsbC5hbGwudHMkZGF0ZSkpLCBtb250aChtaW4obWFpbi5hbGwuYWxsLnRzJGRhdGUpKSwgZGF5KG1pbihtYWluLmFsbC5hbGwudHMkZGF0ZSkpKSkNCg0KIyMgQWx0ZXJuYXRlOiBTZWUgTm90ZXMgYmVsb3cgZm9yIGV4cGxhbmF0aW9uIG9mIGZyZXF1ZW5jeQ0KdHMuTWFpbi5hbGwuYWxsLndrIDwtIHRzKG1haW4uYWxsLmFsbC50cyR2aWV3cywgZnJlcXVlbmN5PTUyLCBzdGFydD1jKHllYXIobWluKG1haW4uYWxsLmFsbC50cyRkYXRlKSksIG1vbnRoKG1pbihtYWluLmFsbC5hbGwudHMkZGF0ZSkpLCBkYXkobWluKG1haW4uYWxsLmFsbC50cyRkYXRlKSkpKQ0KDQpgYGANCg0KTm90ZXM6DQoNCiogc2luY2UgaXQgaXMgZGFpbHkgZGF0YSwgdGltZSBzZXJpZXMgZnJlcXVlbmN5PTM2NSAoZGF5cyBpbiB5cikNCiogaG93ZXZlciwgYmVjYXVzZSB0aGVyZSBpcyB0b3RhbCA1NTAgZGF0YSBwb2ludHMsIG5vdCBlbm91Z2ggZGF0YSBmb3Igc2Vhc29uYWxpdHkgY29tcG9uZW50IHRvIGJlIGVzdGltYXRlZCAobmVlZCBhdCBsZWFzdCB0d28gaXRlcmF0aW9ucykNCiAgICArIHdpbGwgZ2V0IGVycm9yOiAidGltZSBzZXJpZXMgaGFzIG5vIG9yIGxlc3MgdGhhbiAyIHBlcmlvZHMiIg0KKiBzby4uLnVzZWQgZnJlcXVlbmN5PTUyIHRvIHRyZWF0IHRoZSB0aW1lIHNlcmllcyBhcyB3ZWVrbHkgZGF0YSBmb3IgaWxsdXN0cmF0aW9uIHB1cnBvc2VzDQoqIChncmVhdCBmb3IgaWxsdXN0cmF0aXZlIHB1cnBvc2VzIGJ1dCBtaWdodCBiZSBoYXJkIHRvIGdldCBhIHJlbGlhYmxlIHByZWRpY3Rpb24pDQoNCk1vcmUgb24gJ3RpbWUgc2VyaWVzIGhhcyBubyBvciBsZXNzIHRoYW4gMiBwZXJpb2RzJyBlcnJvcjogPGJyIC8+DQoqIGh0dHBzOi8vc3RhdC5ldGh6LmNoL3BpcGVybWFpbC9yLWhlbHAvMjAxMy1PY3RvYmVyLzM2MTA0Ny5odG1sDQoNCiMjIyMgRGVjb21wb3NpdGlvbg0KDQpVc2luZyAnZGVjb21wb3NlJw0KDQpgYGB7ciBUSU1FIFNFUklFUyBNQUlOIERFQ09NUH0NCmRlY29tcG9zZWRSZXMgPC0gZGVjb21wb3NlKHRzLk1haW4uYWxsLmFsbC53aywgdHlwZT0nYWRkaXRpdmUnKSAjIyB0eXBlPSdtdWx0JyBpZiBtdWx0aXBsaWNhdGl2ZTsgJ2FkZGl0aXZlJyBpZiBhZGRpdGl2ZQ0KcGxvdChkZWNvbXBvc2VkUmVzKQ0KDQpgYGANCg0KVXNpbmcgJ3N0bCcNCg0KYGBge3IgVElNRSBTRVJJRVMgTUFJTiBTVEx9DQoNCnN0bC5zdHlsZSA8LSBzdGwodHMuTWFpbi5hbGwuYWxsLndrLCBzLndpbmRvdz0ncGVyaW9kaWMnKQ0KcGxvdChzdGwuc3R5bGUpDQoNCmBgYA0KDQojIyMjIFByZWRpY3Rpb24NCg0KaHR0cHM6Ly9hLWxpdHRsZS1ib29rLW9mLXItZm9yLXRpbWUtc2VyaWVzLnJlYWR0aGVkb2NzLmlvL2VuL2xhdGVzdC8NCg0KUmV2ZXJ0aW5nIGJhY2sgdG8gb3JpZ2luYWwgZGF0YSBhbmQgZm9yZ2luZyBhaGVhZCB3aXRoIHByZWRpY3Rpb24uDQoNCmBgYHtyIFBSRURJQ1RJT04gMn0NCiMjIGJhY2sgdG8gZGFpbHkgZGF0YSBldmVuIHRob3VnaCBsZXNzIHRoYW4gMiBwZXJpb2RzDQp0cy5NYWluZm9yZWNhc3QgPC0gSG9sdFdpbnRlcnModHMuTWFpbiwgYmV0YT1GQUxTRSwgZ2FtbWE9RkFMU0UpDQp0cy5NYWluZm9yZWNhc3QNCmBgYA0KDQoqIGFscGhhIGNsb3NlIHRvIDEgbWVhbnMgaGVhdnkgd2VpZ2h0aW5nIG9uIHJlY2VudCAobmVlZHMgY29uZmlybWF0aW9uKQ0KDQpgYGB7cn0NCnBsb3QodHMuTWFpbmZvcmVjYXN0KQ0KYGBgDQoNCiogcGxvdHRpbmcgZm9yZWNhc3QgYWdhaW5zdCBhY3R1YWwNCg0KIyMjIyBGb3JlY2FzdA0KDQpgYGB7cn0NCiMjIHJlcXVpcmVzIA0KbGlicmFyeShmb3JlY2FzdCkNCmxpYnJhcnkoc3RhdHMpDQp0cy5NYWluZm9yZWNhc3QyIDwtIGZvcmVjYXN0KHRzLk1haW5mb3JlY2FzdCwgaD00NSkgIyMgZG9uJ3QgbmVlZCBmb3JlY2FzdC5Ib2x0V2ludGVycygpDQp0cy5NYWluZm9yZWNhc3QyDQoNCnBsb3QodHMuTWFpbmZvcmVjYXN0MikNCg0KDQpgYGANCg0K